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Abstract. Rank minimization has attracted a lot of attention in recent 
years due to its robustness in data recovery. To overcome the computa- 
tional difficulty, rank is often replaced with its convex surrogate, nuclear 
norm. For several rank minimization problems (RMPs), such a replace- 
ment has been theoretically proven to be valid, i.e., the solution to nu- 
clear norm minimization problem (NNMP) is also the solution to RMP. 
Although it is easy to believe that such a replacement may not always 
be valid, no concrete example has ever been found. We argue that such a 
^"^ validity checking cannot be done by numerical computation and show, by 

£C) analyzing the noiseless latent low rank representation (LatLRR) model, 

^SJ that even for very simple RMPs the validity of replacing rank with nu- 

clear norm may still break down. So a practitioner should not take for 
granted to replace rank with nuclear norm. As a by-product, we find that 
the solution to the NNMP formulation of LatLRR is non-unique. Hence 
the results of LatLRR reported in the literature may be questionable. 
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1 Introduction 



We are now in an era of big data as well as high dimensional data. Fortu- 
nately, high dimensional data are not unstructured. Usually, they lie near low 
(■■<■) dimensional manifolds. This is the basis of linear and nonlinear dimensionality 

C<^ reduction [I]. As a simple yet effective approximation, linear subspaces are usu- 

Cn ally adopted to model the data distribution. Because low dimensional subspaces 

^*-~ correspond to low rank data matrices, rank minimization is now widely used in 

machine learning and data recovery 03 4 5 J . Actually, rank is regarded as a spar- 

sity measure for matrices [3] . So low rank recovery problems are studied |6)7)8|9) 

in parallel to the compressed sensing theories for sparse vector recovery. Typical 

. . rank minimization problems include matrix completion |2I4) . robust principal 

component analysis [3J, and low rank representation |10lllj . which have found 

wide applications, such as background modeling [3], image repairing [T3], im- 

$_i age alignment [12], image rectification [13], motion segmentation |10|llj . image 

segmentation [14] , and saliency detection [15] . 

Since the rank of a matrix is discrete, rank minimization problems (RMPs) 
are usually hard to solve. They can even be NP hard [3]. To overcome the compu- 
tational obstacle, as a common practice people usually replace rank with nuclear 
norm, which is the sum of singular values and is the convex envelope of rank 
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on the unit ball of matrix operator norm [5], to transform RMPs into nuclear 
norm minimization problems (NNMPs). Such a strategy is widely adopted in 
most rank minimization problems [2 3 4110111112113114115] , However, this natu- 
rally brings a question about whether replacing rank with nuclear norm is valid, 
i.e., the solution to an NNMP is also a solution to the corresponding RMP. A 
similar problem has been considered by some scholars. For example, Candes et 
al. proved that when the rank of optimal solution is sufficiently low and the 
missing data is sufficiently few or the corruption is sufficiently sparse, solving 
NNMPs of matrix completion [J] or robust PCA problems [3] can exactly re- 
cover the ground truth low rank solution with an overwhelming probability. Liu 
et al. |10ll6j proved that when the rank of optimal solution is sufficiently low 
and the percentage of corruption does not exceed a threshold, solving the NNMP 
of low rank representation (LRR) |10|11) can exactly recover the ground truth 
low rank solution. However, we want to highlight the difference between our re- 
placement validity problem (RVP) and the exact recovery problem (ERP) that 
scholars have considered before. The RVP is to compare the solutions between 
two optimization problems, while ERP is to study whether solving an NNMP 
can exactly recover a ground truth low rank matrix. As a result, in all the exist- 
ing ERPs, the scholars have to assume that the rank of ground truth solution is 
sufficiently low. In contrast, the RVP does not rely on this assumption: even if 
the ground truth low rank solution cannot be recovered, we can still investigate 
whether the solution to an NNMP is also the solution to the corresponding RMP. 

For RVP, it is easy to believe that for complex RMPs the replacement of rank 
with nuclear norm will break down. While for ERPs, the existing analysis all 
focuses on relatively simple RMPs, such as matrix completion 0, robust PCA 
problems [3], and LRR |10|llj . and has achieved affirmative results under some 
conditions. So it is also easy to believe that for simple RMPs the replacement of 
rank with nuclear norm will work. This paper aims at breaking such an illusion. 
Here, we have to point out that RVP cannot be studied by numerical experi- 
ments. This is because: 1. rank is sensitive to numerical errors. Without prior 
knowledge, one may not correctly determine the rank of a given matrix, even if 
there is a clear drop in its singular values; 2. it is hard to verify whether a given 
solution to NNMP is a global minimizer to an RNP, whose objective function is 
discrete and non-convex. So we should study RVP by purely theoretical analysis. 
We analyze a simple RMP, noiseless latent LRR (LatLRR) [T7], to show that 
solutions to an NNMP may not be solutions of the corresponding RMP. So a 
practitioner should not take for granted to replace rank with nuclear norm. 

The contributions of this paper include: 

1. We use a simple RMP, noiseless LatLRR, to prove that solutions to an 
NNMP may not be solutions of the corresponding RMP, even for very simple 
RMPs. 

2. As a by-product, we find that LatLRR is not a good mathematical model 
because the solution to its NNMP formulation is non-unique. So the results 
of LatLRR reported in the literature, e.g., |17ll0j . may be questionable. 
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2 Latent Low Rank Representation 

In this section, we first explain the notations that will be used in this paper and 
then introduce latent low rank representation which we will analyze its closed 
form solutions. 



2.1 Summary of Main Notations 

A large amount of matrix related symbols will be used in this paper. Capital 
letters are used to represent matrices. Especially, / denotes the identity matrix 
and is the all-zero matrix. The entry at the ith row and the jth column of 
a matrix is denoted by [-]ij- Nuclear norm, the sum of all the singular values 
of a matrix, is denoted by 1 1 • 1 1 * . Operator norm, the maximum singular value, 
is denoted by || • ||2- Trace(^4) represents the sum of the diagonal entries of A 
and A^ is the Moore-Penrose pseudo-inverse of A. For simplicity, we use the 
same letter to present the subspace spanned by the columns of a matrix. The 
dimension of a space V is presented by dim(V). The orthogonal complement 
of V is denoted by V±. Range(^4) indicates the linear space spanned by all the 
columns of matrix A, while Null(A) represents the null space of A. They are 
closely related: (Range(A))^ = Null(A T ). Finally, we always use Ux^xVx to 
represent the skinny SVD of the data matrix X. Namely, the numbers of columns 
in Ux and Vx are both rank(A') and Ex consists of all the non-zero singular 
values of X, making Ex invertible. 

2.2 Low Rank Subspace Clustering Models 

Low rankness based subspace clustering stems from low rank representation 
(LRR) [10111) . The mathematical model of the original LRR is 

minrank(Z), s.t. X = XZ, (1) 

where X is the data matrix we observe. LRR is to use data to express themselves 
and finds the lowest rank representation matrix. It extends sparse subspace clus- 
tering |18j by generalizing the sparsity from ID to 2D. When there is noise or 
corruption, a noise term can be added to the model (1011 1] . Since this paper 
considers closed form solutions for noiseless models, to save space we omit the 
noisy model. The corresponding NNMP formulation of (IT]) is 

min||Z||*, s.t. X = XZ, (2) 

which we call the relaxed LRR. LRR has been very successful in clustering data 
into subspaces robustly [19] . It is proven that when the underlying subspaces 
are independent, the optimal representation matrix is block diagonal, each block 
corresponding to a subspace [lOlllj . 

LRR works well only when the samples are sufficient. This condition may 
not be fulfilled in practice, particularly when the dimension of samples are large. 
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To resolve this issue, Liu et al. [17] proposed latent low rank representation 
(LatLRR) . Another model to overcome this drawback of LRR is fixed rank rep- 
resentation |5D]. LatLRR assumes that the observed samples can be expressed 
by themselves together with unobserved ones: 

minrank(Z), s.t. X = [X,X H ]Z. (3) 

where Xh is the unobserved samples for supplementing the shortage of the 
observed ones. Since Xh is unobserved, by some deduction and mathematical 
approximation, LatLRR [17] is modeled as follows 

minrank(Z) + rank(i), s.t. X = XZ + LX. (4) 

Both the optimal Z and L can be utilized for learning tasks: Z can be used for 
subspace clustering, while L is for feature extraction, thus providing us with the 
possibility for integrating two tasks into a unified framework. We call Q the 
original LatLRR. Similarly, it has an NNMP formulation 

min||Z||„ + ||L||», s.t. X = XZ + LX, (5) 

which we call the relaxed LatLRR. LatLRR has been reported to have better 
performance than LRR |17ll0j . 



3 Analysis on LatLRR 

In this section, we provide surprising results: both the original and relaxed 
LatLRR have closed form solutions! We are able to write down all their so- 
lutions, as presented in the following theorems. 

Theorem 1. The complete solutions to the original LatLRR problem Q are as 
follows 

Z* = V x WV% + SiWV£ andL* = U X S X (L ~W)S x 1 U] l + U X S X (I -W)S 2 , 

(6) 
where W is any idempotent matrix and Si and S2 are any matrices satisfying: 
1. V£ Si = and SJUx — 0; and 2. rank(Si) < rank(W) and rank{S2) < 
rank(I — W) . 

Theorem 2. The complete solutions to the relaxed LatLRR problem ([5]) are as 
follows 

Z* = V X WV X and L* =U X (I- W)U^, (7) 

where W is any block diagonal matrix satisfying: 1. its blocks are compatible 
with S x , i.e., if [S x ]u 7^ [£x\jj then [W]ij = 0; and 2. both W and L —W are 
positive semi- definite. 
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By Theorems |l] and |2j we can conclude that if the W in Theorem [2J is not 
idempotent, then the corresponding (Z*.L*) is not the solution to the original 
LatLRR, due to the following proposition: 



Proposition 1. If the W in Theoremvais not idempotent, then Z* = VxWV^ 
cannot be written as Z* = VxWV^ + S\WVjr , where W and S\ satisfy the 
conditions stated in Theorem^ 

The above results show that for noiseless LatLRR, nuclear norm is not a valid 
replacement of rank. As a by product, since the solution to the relaxed LatLRR 
is non-unique the results of LatLRR reported in |17lll| may be questionable. 

We provide detailed proofs of the above theorems and proposition in the 
following section. 

4 Proofs 

4.1 Proof of Theorem Q] 

We first provide the complete closed form solutions to the original LRR in a 
more general form 

minrank(Z), s.t. A = XZ, (8) 

where A £ Range(X) so that the constraint is feasible. We call ffity the generalized 
original LRR. Then we have the following proposition. 

Proposition 2. Suppose Ua^aVJ is the skinny SVD of A. Then the minimum 
objective function value of the generalized LRR problem ([8]) is rank(A) and the 
complete solutions to (T8J) are as follows 

Z*=X^A + SVj, (9) 

where S is any matrix such that V^S = 0. 

Proof. Suppose Z* is an optimal solution to problem J8]). First, we have 

rank(^l) = rank(ATZ*) < rank(Z*). (10) 

On the other hand, because A = XZ is feasible, there exists Z\ such that 
A = XZ X . Then Z„ = X^A is feasible: XZ Q = XX^A = XX^XZ Y = XZ X = A, 
where we have utilized a property of Moore- Penrose pseudo-inverse XX^ X = X . 
So we obtain 

rank(Z*) < rank(Z ) < rank(A). (11) 



Combining (10 1 with (111, we conclude that rank(A) is the minimum objective 
function value of problem $h . 

Next, let Z* = PQ T be the full rank decomposition of the optimal Z* , 
where both P and Q have rank(A) columns. From Ua^aVJ = XPQ T , we 
have Vj[ = (S^U^X P)Q T . Since both Va and Q are full column rank and 
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Y = S^U^XP is square, Y must be invertible. So V A and Q represent the same 
subspace. Because P and Q are unique up to an invertible matrix, we may simply 
choose Q = V A . Thus U A S A Vj = XPQ T reduces to U A E A = U X £ X V£P, i.e., 
V X P — S^fXJ^JJ a^ Uj an d we conclude that the complete choices of P are 
given by P — V x £ x U x Ua£a + S, where S is any matrix such that V x S = 0. 
Multiplying P with Q T — V A , we obtain that the entire solutions to problem (fsl) 
can be written as Z* = X'A + SV A , where S is any matrix satisfying V x S = 0. 
□ 

Similar to Proposition [21 we can have the complete closed form solution to 
the following problem 

minrank(L), s.t. A = LX, (12) 

which will be used in the proof of Theorem [TJ 

Proposition 3. Suppose U a S a Va * s ^ e skinny SVD of A. Then the minimum 



objective function value of problem ( 12 ) is rank(A) and the complete solutions 



to problem (12 I are as follows 

L* = AX^ + U A S, (13) 

where S is any matrix such that SUx = 0. 

Next, we provide the following propositions. 

Proposition 4. rank(X) is the minimum objective function value of the original 
LatLRR problem Q . 

Proof. Suppose (Z* , L*) is an optimal solution to problem Q. By Proposition [2] 
and fixing Z* , we have rank(L*) = rank(X — XZ*). Thus 

rank(Z*) + rank(L*) > xscak(XZ*) + rank(X - XZ*) > rank(X). (14) 

On the other hand, if Z* and L* are adopted as X^X and 0, respectively, the 
lower bound is achieved and the constraint is fulfilled as well. So we conclude 
that rank(X) is the minimum objective function value of the original LatLRR 
problem Q. □ 

Proposition 5. Suppose (Z*,L*) is one of the solutions to problem Q. Then 
there must exist another solution (Z, L), such that XZ* = XZ and Z = VxWV x 
for some matrix W . 

Proof. According to the constraint of problem Q, we have XZ = {I— L)X, i.e., 
{XZ) T £ Range(X T ). Since V x V x is the projection matrix onto Range(X T ), 
we have 

XZ*VxV x =XZ*. (15) 

On the other hand, given the optimal Z* , L* is the optimal solution to 

minrank(i) s.t. X(I - Z*) = LX. (16) 
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So by Proposition [2] we get 

rank(L*) = rmk(X(I - Z*)X^). (17) 

As a result, 

rank(X) = rank(Z*) + rank(L*) 

= rank(Z*) + rank(AT(/ - Z*)X r ) 

= rank(Z*) + iank(X(I - V X V% Z*V X V£)X^) (18) 

> Taa3k(V x VgZ*V x Vx) + mnk{X(I - V X V£ Z*V X V£)X^) 

> rank(X), 

where the last inequality holds since (y x VgZ*V x V%, X(I-V x V$Z*V x V$)Xl) 
is a feasible solution to problem Q and rank(X) is the minimum objective ac- 
cording to Proposition^ @ shows that (V X V^Z*V X V^, X{I-V X V%Z*V X V£)X^) 
is an optimal solution. So we may take Z — V X V X Z*V X V X and write it as 
Z = VxWVg, where W = V%Z*V X . 

Finally, combining with equation (15 1, we conclude that 

XZ = U x ExVxV x VgZ*VxVx = XZ*V X V X = XZ*. (19) 

n 

Proposition [5] provides us with a great insight into the structure of problem 
0): we may break Q into two subproblems 



z 

and 



minrank(Z), s.t. XV X WV£ = XZ, (20) 



mm 
L 



rank(L), s.t. X - XV X WV£ = LX, (21) 



and then apply Propositions [2] and [3] to find the complete solutions to problem 

©■ 

For investigate the properties of W in (20) and (21 1, the following lemma is 

critical. 

Lemma 1. For A, B 6 M. nxn , if AB = BA, then the following inequality holds 

rank(A + B) < rank(A) + rank(B) - rank(AB). (22) 

Proof. On the basis of AB = BA, it is easy to check that 

Null(A) + Null(B) c Null(AB), (23) 

and 

Null(A) n Null(B) c Null(A + B) . (24) 
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On the other hand, according to the well-known dimension formula 

dim(Null(A))+dim(Null(5)) = dim(Null(A)+Null(-B))+dim(NuU(A)nNull(S)), 

(25) 



by combining ( 25 ) with ( 23 ) and (p4| , we get 



dim(Null(A)) + dim(Null(5)) = dim(Null(AB)) + dim(Null(A + B)). (26) 

Then by the relationship rank(S') = n — dim(Null(S')) for any S G R nx ™, we 
arrive at the inequality (|22J). D 



Based on the above lemma, the following proposition presents the sufficient 
and necessary condition on W . 



Proposition 6. LetL* be any optimal solution to subproblem (21), then (VxWX x ,L*) 
is optimal to problem (Hi) if and only if the square matrix W is idempotent. 



is 



Proof. Obviously, (VxWX x ,L*) is feasible based on the constraint in problem 



( 2l|). By considering the optimality of L* for (|2l|) and replacing Z* with VxWVx 



in equation (17), we have 

rank(L*) = rank(X(7 - V X WV%)X^). (27) 

First, we prove the sufficiency. According to the property of idempotent ma- 
trices, we have 

rank(W') = trace(W') and rank(7 - W) = trace(7 - W). (28) 

By substituting (VxWVx,L*) into the objective function, the following equali- 
ties hold 

mnk{V x WV£) + rank(L*) = rank(VJ/) + rank(X(7 - V X WV% )X^) 

= rank(FT) + nmk(U x £x(I - W)£ x lu x) 

= rank(Vi/) + rank(7 - W) (29) 

= trace(W>) + trace(J - W) 

= rank(X). 

So (VxWXx,L*) is optimal since it achieves the minimum objective function 
value of problem BJ . 

Second, we prove the necessity. Suppose {VxWV x , L*) is optimal to problem 
PJ. Substituting it into the objective follows 

rank(X) = nmk(V x WX x ') + rank(X(7 - V X WV X )X^) 

= rank(VJ/) + rank(7 - W) (30) 

> rank(X). 

Hence rank(iy) + rank(I — W"l_ = ramc PQ- On the other hand, as W and I —W 
axe commutative, by Lemma 111 we have rank(X) < rank(iy) + rank(7 — W) — 
va,nk(W - W 2 ). So rank(H> - W 2 ) = and thus W = W 2 . D 
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We are now ready to prove Theorem fT] 

Proof. Solving problems p0| and ( plj ) by using Propositions I2I and pi where W 
is idempotent as Proposition [6] shows, we directly get 

Z* = V x WVx + S-iVX and L* = U X S X (I - W)E x l llJr + U B S 2l (31) 

where U A S A Vj and U B S B Yb are the skinny SVDs of U x £ x WVg and U X S X {I- 
W)V X , respectively, and Si and S 2 are matrices such that V X S\ = and 
S 2 U X = 0. Since we have Range ((WV£) T ) = Range(V^) and Range ( U x S x (I - 
W)) — Range(t/s), there exist full column rank matrices Mi and M 2 satisfying 
V A = {WV^) T M 1 and U B = U X £ X {I-W)M 2 , respectively. The sizes of Mi and 
M 2 are rank(X) x rank(M / ) and rank(X) x rank(I —W), respectively. We can eas- 
ily see that a matrix Si can be decomposed into Si = Si M-f , such that V X S\ = 
and Mi is full column rank, if and only if V^Si = and rank(Si) < rank(W^). 
Similarly, a matrix S 2 can be decomposed into S 2 = M 2 S 2 , such that S 2 U X = 
and M 2 is full column rank, if and only if S 2 U X = and rank(S2) < rank(7— W). 
By substituting V A = (WV$) T Mi, U B = U X S X (I-W)M 2 . Si = SiMf , and 
S 2 = M 2 S 2 into (31 1, we obtain the conclusion of Theoremlll □ 



4.2 Proof of Theorem [2] 

We first quote two results from [TU] . 

Lemma 2. Assume X =/= and A = XZ have feasible solution(s), i.e., A G 
Range(X). Then 

Z* = lU (32) 



is the unique minimizer to the generalized relaxed LRR problem: 

minllZlL, s.t. A = XZ. 
z 



(33) 



Lemma 3. For any four matrices B, C ' , D and F of compatible dimensions, we 
have the inequalities 



BC 
D F 



> \\B\L 



\F\\* and 



BC 
D F 



>\\B\U, 



(34) 



where the second equality holds if and only if C = 0, D = 0, and F = 0. 

Then we prove the following lemma. 

Lemma 4. For any square matrix Y € M. nxn , we have ||V||* > trace(Y), where 
the equality holds if and only if Y is positive semi-definite. 

Proof. We prove by mathematical induction. When n = 1, the conclusion is 
clearly true. When n = 2, we may simply write down the singular values of Y 
to prove. 
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Now suppose for any square matrix Y, whose size does not exceed n — 1, the 
inequality holds. Then for any matrix Y € M™ xn , using Lemma |3j we get 



\Y\ 



Yu Y\2 

Y21 Y 2 2 

> \\Yn\U + \\Y22\U (35) 

> trace(Yn) + trace(l22) 
= trace(Y), 

where the second inequality holds due to the inductive assumption on the ma- 
trices Yu and Y 2 2- So we always have ||Y||* > trace(Y). 

It is easy to check that any positive semi-definite matrix Y, it satisfies ||Y| |* = 
trace(F). On the other hand, just following the above proof by choosing F 2 2 
as 2 x 2 submatries, we can easily get that ||Y||* > trace(y) strictly holds if 
Y £ M. nxn is asymmetric. So if ||F||* = trace(V), then Y must be symmetric. 
Then the singular values of Y are simply the absolute values of its eigenvalues. 
As trace(y) equals the sum of all eigenvalues of Y, ||Y||* = trace(y) holds only 
if all the eigenvalues of Y are non-negative. □ 

Using Lemma [2| we may consider the following unconstrained problem 

mmf(Z)±\\Z\\* + \\X(I-Z)Xl\U, (36) 

which is transformed from |5]) be eliminating L therein. Then we have the fol- 
lowing result. 



Proposition 7. Unconstrained optimization problem (36) has a minimum ob- 
jective function value rank(X). 

Proof. Recall that the sub-differential of the nuclear norm of a matrix Z is [3T] 

d z \\Z\U = {UzVl + R\U%R = 0,RV z = 0, ||fi|| a < 1}, (37) 

where U Z £ Z V£ is the skinny SVD of the matrix Z. We prove that Z* = \J2X^X 



is an optimal solution to (36 1. It is sufficient to show that 



e d z f(Z*) = d z \\Z*\U + d z \\X(I - Z*)X% 

= d z \\Z*\\. - X T d xiI _ z)xi \\X(I - Z*)Xt||,(Xt)T 

Notice that X(I - Z*)X^ = U X (1/2I)U X ' is the skinny SVD of X(I - Z*)X^ 
and Z* = V X (1/2I)V£ is the skinny SVD of Z* . So d z f(Z*) contains 

V x Vl - X T {VxV T x ){X^f = v x vl - V x Z x Vl\J x \JlU x X x x Vl = 0. (39) 



Substituting Z* = 1/2X'X into (36), we get the minimum objective function 
value rank(V). D 



Next, we have the form of the optimal solutions to (36) as follows 
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Proposition 8. The optimal solutions to the unconstrained optimization prob- 



lem ( 36 ) can be written as Z* 



VxWV£. 



Proof. Let (Vx)± be the orthogonal complement of V x . According to Propo- 
sition [7l rank(X) is the minimum objective function value of (36). Thus we 
get 



rank(AT) = \\Z* 



v$ 



X(I -Z*)X*\\ 

Z*[V x ,(Vx)±] 



(Vx)l 

V£Z*V X V^Z*(V x )i_ 
(V x )lZ*V x (Vx)lZ*(V x ) 



+ \\X(I-Z*)X^\\ t 

+ \\x(i-z*)x^\u 



> \\v$z*v x \\* + \\u x z x v£(i- z*)v x s x lu l\\* 

= \\V X VZZ*V X VZ\\* + \\UxZxVZ{I-VxV£z*VxV£)Vx£ x l Ux'\\* 
= \\VxV£z*VxV£\U + \\X(I-VxV£z*VxV£)Xl\\* 
> rank(X), 

(40) 

where the second inequality holds by viewing Z = V X V X Z*V X V X as a feasible 
solution to (36). Then all the inequalities in (40) must be equalities. By Lemma 
[3]we have 



V X Z*(V X )± = (Vx)iZ*V x = (v x y ± z*(v x )± = 



That is to say 



V x 

(Vx)l 



Z*[V x ,(Vx)±] = 



WO 




where W — V x Z*Vx- Hence the equality 



Z* 



[VxAV 



XI 



wo 





vl 

(Vx)l 



VxWV\ 



x 



(41) 
(42) 

(43) 



holds. □ 



Based on all the above lemmas and propositions, the following proposition 
gives the whole closed form solutions to the unconstrained optimization problem 
(36). So the solution to problem (36) is non-unique. 



Proposition 9. The solutions to the unconstrained optimization problem (36) 
are Z* — VxWVl , where W satisfies: 1. it is block diagonal and its blocks are 
compatible with SxYJ %■ both W and I — W are positive semi- definite. 



1 Please refer to Theorem 2 for the meaning of "compatible with Ex" 
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Proof. First, we prove the sufficiency. Suppose Z* — VxWVx satisfies all the 
conditions in the theorem. Substitute it into the objective function, we have 

til - 



\Z*\\, + \\X(I-Z*)X^\\, = 



= \\ w \ 


* + 


\\Ex(I- 


W)E X % 


= II wi 


* + 


trace (Ex 


[I-W)E X 1 ) 


= \\w\ 


* + 


trace(I — 


W) 


= \\w\ 


* + 


rank(X) 


- trace(W / ) 


= rank(X) 






= min 

z 


\z\\ 


. + IW 


-Z)X f lk 



(44) 



where based on Lemma |4J the second and the fifth equalities hold since I ~W — 
Ex (I — W)E X as W is block diagonal and both I — W and W are positive 
scmi-dcfinitc. 

Next, we give the proof of the necessity. Let Z* represent a minimizer. Ac- 
cording to Proposition ^l Z* could be written as Z* = VxWV x ■ We will show 
that W satisfies the stated conditions. Based on Lemma |4j we have 

rank(X) = 

\\w\\* + \\e x (i-w)e x % 

(45) 



= 11^*11, 


+ \\x(i- 


Z*)Ji 


' f ll* 


= \\w\u 


+ \\E X (I- 


-W). 


-'X II* 


>\\w\\. 


+ trace(E x (I- 


W)E X X 


= \\w\U 


+ trace (J - 


-W) 




= \\w\\. 


+ rank(X) 


- trace(W") 


> rank(X). 







Thus all the inequalities above must be equalities. From the last equality and 
Lemma|4j we directly get that W is positive semi-definite. By the first inequality 
and Lemma kl we know that Ex (I — W)E X 1 is symmetric, i.e., 

^{I-W] ij = ^[I-W} ij , (46) 

where ai represents the ith entry on the diagonal of Ex ■ Thus if c, =^ (Xj , then 
[I — W]ij — 0, i.e., W is block diagonal and its blocks are compatible with Ex- 
Notice that I-W = E X (I - W)E x l . By LemmajiJ we get that I - W is also 
positive semi-definite. Hence the proof is completed. □ 

Now we can prove Theorem [2] 

Proof. Let W satisfy all the conditions in the theorem. According to Proposition 
^l since the row space of Z* = V X WV X belongs to that of X, it is obvious that 
(Z* , X(I — Z*)X^) is feasible to problem ^. Now suppose that ^ has a better 
solution (Z, L) than (Z*,L*), i.e., 

X = XZ + LX, (47) 
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and 

||£||* + ||£||* < \\Z*\\, + \\L*\\,. (48) 

Fixing Z in (I5| and by Lemma pi we have 

\\Z\U + \\(X-XZ)X^\U<\\Z\U + \\L\\. (49) 

Thus 

||% + \\(X - XZ)X*\\, < \\Z*\\* + \\X(I-Z*)X*\\*. (50) 

So we obtain a contradiction with respect to the optimality of Z* in Proposition 
[9j hence proving the theorem. □ 

4.3 Proof of Proposition [l] 

Proof. Suppose the optimal formulation Z* = VxWVx in Theorem ^ could be 
written as Z* = VxWVx + SiWVx, where W is idempotent and Si satishes 
WVgSi = 0. Then we have 

V X WX X = V X WV$ + SiWVx. (51) 

By multiplying both sides with V x and Vx on the left and right, respectively, 
we get 

W = W + V^SiW. (52) 

As a result, W is idempotent: 

W 2 = (W + V^SiW){W + VgSiW) 

= W 2 + VxSiW 2 + WVxSiW + VxSiWVxSiW 

= W + VgSiW + WVxSiW + VxSiWVxSiW 

= W + VgSiW = w, 

which is contradictory to the assumption. □ 

5 Conclusions 

Based on the expositions in Section [3] and the proofs in Section |4j we conclude 
that even for RMPs as simple as noiseless LatLRR, replacing rank with nuclear 
norm is not valid. So a practitioner should not take such a replacement in RMPs 
for granted. We have also found that LatLRR is actually problematic because 
the solution to its NNMP formation is not unique. We can also have the following 
interesting connections between LRR and LatLRR. Namely, LatLRR is indeed 
an extension of LRR because its solution set strictly includes that of LRR, no 
matter for the RMP or the NNMP formulation. So we can summarize their 
relationship as Figure [T] 

Although the existing formulation of LatLRR is imperfect, since some schol- 
ars have demonstrated its effectiveness in subspace clustering by using a solution 
which is randomly chosen in some sense, in the future we will consider how to 
choose the best solution in the solution set in order to further improve the per- 
formance of LatLRR. 



(53) 
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Original LRR (1) 



extension 



valid surrogate 



Original LatLRR (4) 



invalid surrogate 



Relaxed LRR (2) 



extension 



Relaxed LatLRR (5) 



Fig. 1. The detailed relationship among the original LRR |T]), the relaxed LRR pj, 
the original LatLRR Q, and the relaxed LatLRR (T5J) in the sense of their solution 
sets. 
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